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ABSTRACT 



The National Assessment of Adult Literacy (NAAL) , which is 



being developed for administration in 2002, will determine the status of 
adult literacy in the United States and serve as a follow-up to the 1992 
National Adult Literacy Survey. Three discussion groups consisting of a total 
of 23 literacy researchers, users of the 1992 survey, potential users of the 
2002 survey, and other stakeholders were convened to obtain input on the 
NAAL's content and issues related to its development. A professional 
facilitator used a set of structured questions and probes to lead the 
discussions with each group. Several problems in using the data from the 1992 
survey at the federal and state levels were noted. Suggestions for making the 
NAAL more meaningful than the 1992 survey were quite similar among the three 
groups. A major focus was on ensuring that the data could be useful at the 
state level even if states cannot conduct an individual survey. Suggestions 
for interpretation of the NAAL fell into four topical areas: use of 
subscales, broadness of levels, utilization of composite scores, and linkages 
between the 2002 assessment and other surveys. (Appended are a list of focus 
group participants and overview of the NAAL.) (MN) 
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Executive Summary 



Developing the National Assessment of Adult Literacy: 

Recommendations from Stakeholders 

The National Center for Educational Statistics (NCES) of the Office of Educational Research 
and Improvement (OERI) is planning a National Assessment of Adult Literacy, to be conducted in 
2002. This survey will determine the status of adult literacy in the United States and will serve as a 
follow-up to the 1992 National Adult Literacy Survey. To obtain input on the content of the survey 
and issues related to the survey development, NCES convened three discussion groups of literacy 
researchers, users of the 1992 survey, potential users of the 2002 survey and other stakeholders. The 
American Institutes for Research (AIR) facilitated these discussions, held in January and February 
1998. 

Organization of the Discussion Groups 

Stakeholders recommended that the next National Assessment of Adult Literacy have the 
following goals: 

❖ To estimate the current status of literacy skills in the U.S. adult population; 

❖ To monitor trends in the status of adult literacy in America; 

❖ To inform federal and state policy; and 

❖ To support and enhance literacy programs at the state level. 

These goals suggest that the study will be addressed to three audiences - federal officials, 
state policymakers and the public, and representatives of the general media (which informs the 
general public) and educational media (which informs researchers and policymakers). 

Consequently, NCES invited prominent representatives from each of these sectors to each meeting 
and a total of 23 people participated in the discussion groups. Appendix A lists the 23 participants. 

Discussion Questions 

A professional facilitator led the discussions with each group, using a set of structured 
questions and probes. Discussion group participants shared their reactions to, and use of, the 1992 
literacy survey; discussed ideas for improving interpretation of the 2002 survey; discussed 
subpopulations to examine in the analysis of the 2002 data; and offered suggestions for improving 
the content, reporting mechanisms, and dissemination strategies of the new assessment. The specific 
discussion questions are listed in Exhibit 1. 
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Organization of this Report 

This document presents summaries of the discussions in two ways. First, an overall 
summary provides a detailed synopsis of the discussions and recommendations from the three groups 
of stakeholders. Following this overall synopsis are individual summaries of each of the three 
discussion groups. Appendix B gives an overview of the current plans for and general issues in 
developing the next National Assessment of Adult Literacy, prepared by Andrew Kolstad of the 
Department of Education. Dr. Kolstad provided this overview at the start of each meeting. 
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Exhibit 1: Discussion Questions 

Reactions to and Use of the 1992 National Adult Literacy Survey 

❖ What problems did you have using information from the 1992 survey? What made it hard to use? 

❖ What aspect of the 1992 survey was easy to present? 

Interpreting the 1992 data 

❖ In what way would the presentation of this material be most useful to you, e.g., standards, grade 
levels, scale scores, composite scores? 

❖ Did you find all of the literacy subscales useful? What scale were/were not useful? 

❖ Would it be helpful, if it were feasible, to link the new assessment to other tests or assessments, 
such as the National Assessment of Education Progress, or other (e.g., state or international-level) 
assessments? 

Sub-Population Study and Analysis 

❖ Were the 1992 surveys of subpopulations, such as the prison population, useful? Did you use any 
of that information? 

❖ What other subpopulation groups do you work with? Would it be helpful to try to provide a 
subpopulation assessment for these groups? 

❖ In the new assessment, are there other analyses of subpopulations you would like to see: e.g., rural 
resident, high school dropouts? Elderly? 

❖ Do you think the 1992 survey adequately addressed the issues related to not native English speaker 
or to bilingual native English speakers? Is this important? What was missed and how would it be 
done better? 

❖ Should the 2002 survey be in English only? 

Designing and Disseminating Findings from the 2002 National Assessment of Adult Literacy 

❖ What should be the goals of the new national assessment of adult literacy? 

❖ What information should be included in the background questionnaire? 

❖ What materials or training in using the 1992 survey data would make it easier for you to use and 
disseminate the material? 

❖ How useful would a new survey be to you? How would you use the information? 

❖ In what areas might discussion papers be useful to help make final decisions regarding the new 
assessment of adult learning? 
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Summary of Focus Group Strategy Sessions 

This summary is a compilation of comments received during three strategy sessions held at 
AIR to provide input to NCES on developing the National Assessment of Adult Literacy. Each group 
was asked the same series of questions, although the depth of discussion on these questions varied 
widely from group to group. The discussion questions were designed to: 

❖ Elicit reactions to the 1992 National Adult Literacy Survey; 

❖ Provide recommendations for the development of a second survey, the National 
Assessment of Adult Literacy, scheduled for 2002. 

Summaries of the individual focus groups follow this overview. 

Each focus group session began with a brief presentation by Andrew Kolstad, Project Office 
from NCES. Dr. Kolstad provided participants with an overview of the project’s goals, assumptions, 
and schedules, as well as a review of unsolved problems based on experience with the previous 
survey. A summary of Dr. Kolstad’s overview is provided as Attachment B. 

Reactions to and Use of the 1992 National Adult Literacy Survey 

The 1992 survey was primarily used to describe the status of literacy to policy makers at the 
federal and state levels, specifically for the purpose of making a case for increased funding and 
support for adult literacy programs. Examples of this usage at the national level include the following: 

❖ The Division of Adult Education and Literacy (DAEL) of the US Department of Education 
utilized the results in reports to Congress and to gain support for special studies, such as a 
welfare analysis of the 1992 survey. 

❖ The National Institute for Literacy (NEFL) used the data to respond to media inquiries 
concerning the size of the illiterate population. 

❖ The Public Broadcasting System used the data to justify a national project to fund distance 
learning for adults. 

❖ The National Education Goals Panel set Level 3 on the three literacy scales as national 
literacy goals and requested another survey to determine progress toward those goals. 

❖ The data were used in speeches focusing on literacy. 

❖ In the private sector, the data were used to encourage publishing companies to invest in 
product development. 

Ability to utilize the data at the state level was more readily available if states had participated 
in one of the State Adult Literacy Surveys (SALS). Thirteen states contracted with ETS for the state 
surveys. Although the Department of Education made synthetic estimates available for all the states. 
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states that had state sample supplements made better use of the instrument. Some examples of that 
usage included the following: 

❖ In New York State, the New York Adult Literacy Survey was used as a needs assessment 
and to argue for education and welfare reform. 

❖ In Illinois, where multiple agencies participated in the survey, questions were added to 
more broadly reflect that state’s needs. The Illinois Adult Literacy Survey provided useful 
information and marked the beginning of a more collaborative effort at data collection. 

State level data were important to Congressional representatives who were interested in knowing how 
their states compared to the national data. 

Although all states that had participated in the initial state adult literacy surveys felt it was 
worthwhile, when asked if they would support state sample supplements for the 2002 survey, state 
directors indicated it would be more difficult to find the funds for another round. 

Problems in Using the 1992 National Adult Literacy Survey 

Several problems were noted in using the 1992 survey data: 

❖ The complexity of the three Literacy scales and the five levels made it difficult to present 
the information to the public in an easily understandable format. The information on the 
literacy continuum needed to be presented in a more user-friendly manner, perhaps 
through descriptions of the populations which are likely to fall into these levels. The 
public relates better to descriptors rather than scales. When the data were presented in a 
simplistic format, however, it masked issues of policy importance. 

❖ Because literacy was so broadly defined it was hard to know what was “enough” literacy. 
A public relations strategy was needed, in advance of the release of the survey, that 
defined literacy for the public and translated the findings in a meaningful way. Without 
this strategy, literacy data often were reported out of context. The Canadian public 
relations model was cited as an example of a quality marketing strategy. 

❖ The difficulty in explaining the five literacy levels led people to interpret the data 
themselves. A lack of clarity and consistency in reporting the data were evident across the 
country. 

♦> A lack of congruence existed between an individual’s self-perception of literacy skills and 
measured literacy skills. The respondents in Levels 1 and 2 did not perceive themselves as 
having a literacy problem. They did not see themselves at risk with a changing labor 
market nor did they believe their skills as inadequate for work and home. 

❖ The lack of release of many of the reports, particularly on the workforce, caused a great 
loss of momentum in reporting results. 
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♦> The Limited English Proficient (LEP) population did not show up clearly in the reports 
and as a result, scores in some states with high percentages of LEP adults showed 
significantly lower scores. 

Making the National Assessment of Adult Literacy More Meaningful 

Suggestions for making the next survey more meaningful were quite similar among the three 
groups. A major focus was on ensuring that the data could be useful at the state level, even if states 
cannot conduct an individual survey. Among the recommendations offered were the following: 

❖ Cross-reference the 2002 survey data with other adult literacy assessments utilized at the 
state level, such as the TABE or ABLE. A broad national sample using a quick checklist 
with vocabulary and correlations would help people cross walk the data. 

❖ Follow the Canadian model and utilize the survey to show what people can do rather than 
what they cannot do. The Canadians have translated their data in a way that is meaningful 
to different aspects of their community, particularly the workforce area, and have 
mobilized the media around these issues. NCES needs to do the same. 

❖ Go beyond numerical scales to more descriptive information regarding the literacy levels 
of diverse populations (e.g. welfare, unemployed, incarcerated). It is important to report 
information that will help administrators manage their programs and resources better. 

❖ Clarify what needs to be reported. There may need to be a ’’quick and dirty” snapshot for 
the public and a more “in-depth” analysis necessary for people who have to work with the 
findings for the purposes of programmatic and policy decisions. 

❖ Find a mechanism to relate the 2002 survey results to the demographic and dynamic 
changes in society. Try to report the information within a broader context. For example, 
look at workforce development efforts and relate information about the skills workers have 
and the skills they need to be proficient in the workforce at various levels — entry level or 
advanced. Similarly, look at parenting skills and report information on the number of 
parents who lack skills to support their child’s educational progress. 

❖ Consider a norm-referenced assessment that shows where people stand in relationship to 
one another, rather than reporting an absolute standard. 

❖ Construct the assessment on sound, validated theory about literacy. The 1992 assessment 
included complex information tasks, and the lack of success may be related to overloaded 
memories, not lack of literacy skills. A suggestion was made to utilize vocabulary tests 
and assess the knowledge that is most likely derived from literacy. 

❖ Crosswalk the data with the literacy levels that are being standardized in the National 
Reporting System. This will be most useful to state program managers. 

❖ Consider timing the release of the reports at the national level to a Congressional calendar. 
Attention to these studies will be most interesting for members of Congress when they are 
reauthorizing the Adult Education Act. 
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* 1 * Continue using data charts demonstrating the relationship between parent’s education and 
children’s level of achievement, as well as charts related to literacy and economic levels 
(e.g. median weekly wages by literacy level, percent of adults in certain categories by 
literacy level). These charts demonstrate the link between earnings and literacy. 

❖ Utilize the World Wide Web to generate interest before, during, and after the survey. 



Interpretation of the Literacy Assessment 

Suggestions for interpretation of the literacy assessment scales fell into four topical areas: use 
of sub-scales, broadness of levels, utilization of composite scores, and linkages between the 2002 
assessment and other surveys. 




❖ Use of Sub-Scales: Although there seemed to be agreement that three scales were already 
difficult to interpret, most discussion focused on increasing the number of scales to include 
such topics as problem solving and team work, oral language skills, technology, and 
writing. It was further suggested that a numeracy scale be developed, to replace the 
quantitative scale. Much discussion focused on the issue that time and money factors need 
to be taken into account. Each time an additional scale is added, costs and time to 
administer the survey increase. It would be necessary to prioritize the scales that present 
the most useful information to the public. 

❖ Participants further suggested that the three current scales, prose, document, and 
quantitative, would only be useful if they could really be differentiated from one another. 

In 1992, it seemed there was a significant overlap among scales. 

❖ Use of Levels: In one of the focus groups, concern was expressed that Level I was too 
broad. It made it difficult to identify the truly non-literate population. 

❖ Composite scores: Two of the groups strongly supported the use of composite scores for 
the prose and document surveys to make it easier to explain the results to the public. The 
third group did not support the use of composite scores; they indicated that utilizing only 
one score might cause teachers to only focus on that one skill. Instead they suggested that 
the scores be related to different aspects of a person’s life, e.g., document scores might be 
related to a person’s employability while prose might be seen in the context of family 
literacy. Participants also noted that different sub-groups perform differently on different 
scales. A composite score would not identify these differences. 



Linkages with other surveys: There was much discussion about providing some sort of 
linkage to other surveys as a basis of comparison. No agreement was reached as to which 
surveys might be appropriate, although most felt that a linkage to the International Adult 
Literacy Survey would be useful. While there did not seem to be much support for a 
linkage between the National Assessment of Adult Literacy and the National Assessment 
of Educational Progress (NAEP), in general, there was some support for a linkage to 
NAEP’s 2002 reading assessment of 12th grade students. 
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Sub-Population Study and Analysis 

Discussion on this topic focused on which, if any, sub populations should be examined, and 
whether some groups should be oversampled. The major discussion points included the following: 

❖ Several groups were identified as needing a sufficient sample size for purposes of public 
policy discussion. These groups were rural adults, adults in welfare-to-work programs, 
adults in low socio-economic circumstances, Native American Indians, Blacks, Hispanics, 
and the elderly. There was much discussion concerning the need to oversample some of 
these groups to ensure adequate representation in the survey. 

❖ Other populations mentioned as possibilities for sampling included individuals with 
disabilities, the population between 16 and 18 years of age, and incarcerated individuals. 

❖ It was determined that a literacy assessment in Spanish was neither practical nor politically 
expedient at this time. 

❖ The project officer indicated that he anticipated a smaller overall sample size for the 2002 
survey. All of the focus group participants urged a larger, not smaller sample, particularly 
to ensure appropriate representation of some of the sub-groups mentioned above. 

Designing and Disseminating Findings from the Next Literacy Assessment 

This section focused on questions related to the goals of the next literacy assessment, the role 
of the background questionnaire, and areas where further discussion and clarification might be needed. 

Goals 

Several similar goals were mentioned in each group. Participants thought the next literacy 

assessment should: 

❖ Provide data to enable administrators to make programmatic and instructional decisions; 

❖ Provide data that is relevant and can be useful to states; 

❖ Indicate changes in literacy over time; and 

❖ Enable us to examine the U.S. population in relation to the rest of the world. 

Background Data 

Participants were very supportive of the project’s assumption that the background 
questionnaire be expanded to provide more descriptive information about those surveyed. Some 
suggestions for additional data included: 

❖ Provide information on the individual’s educational experiences including participation in 
adult basic education and continuing education programs; 
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❖ Provide information on language use and facility with other languages; and 

❖ Provide information on citizenship issues. 

Report Format 

Several comments were made regarding the format for reporting. These comments included: 

❖ Make reports short and simple; 

❖ Provide one page summaries of topical areas for policy makers; 

❖ Utilize charts showing national and state data; 

❖ Consider the design of a state survey that could be more widely implemented; and 

❖ Consider collaborating with other agencies that have the need to know information about 
similar populations. This could enhance funding for the survey as well as provide other 
arenas for marketing and disseminating the data in a meaningful way to diverse audiences. 

Topics for Further Discussion 

The following topics were suggested as possible commissioned papers to help clarify the 
discussion around these topics: 

❖ Effecting interagency collaboration around survey development; 

❖ Cost benefit analysis of a writing assessment; 

❖ Defining the focus of the survey — lower functioning adults or the broader population; 

❖ Interview methodology — the substitution or addition of telephone surveys as part of the 
process; 

i* Standard setting — differentiating between the 80% response probability of the IALS and 
NALS, and that of other surveys including TIMSS and CASAS; and 

❖ Adding additional subpopulations. 



State Survey Development 

Several representatives in the focus groups also had supported the development of individual 
state surveys during the implementation of the 1992 assessment. The State Directors indicated that 
they had participated for a variety of reasons including the need to utilize the data to help direct their 
states’ literacy agendas. 

States funded the survey by utilizing Federal Section 353 dollars, using some state dollars, or 
through collaboration with other state agencies. One state tried to use a local university to duplicate 
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the national process, however, it was costly and ineffective. It was not clear that funding would be 
easily obtained for a second survey. 

While the desire for individual state surveys seemed evident, it was also clear that a single 
assessment contractor could not be utilized because the reports would not be tailored enough to be 
useful for the political needs in each state. Synthetic estimates may be a viable alternative although 
the same interpretation issues remain: states need assistance in interpreting and utilizing the data. 



Usefulness of a Second Survey 

All three focus groups believed a second survey would be useful, particularly in terms of trend 
data. Participants also noted that the 1992 data is “old” and difficult to sell. Finally, if an appropriate 
education and marketing effort were undertaken for the next assessment, media could be more 
supportive and the issue would receive greater attention. 
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Strategy Session: January 27, 1998 

The first strategy session for the purpose of developing a National Assessment of Adult 
Literacy to be conducted in 2002 was held on January 27, 1998 at the offices of Pelavin Research 
Center in Washington, DC. A list of attendees is included in Attachment A. 

The session began with an overview, by Andrew Kolstad, of project goals, assumptions, and 
schedule, as well as unsolved problems. A summary of this overview is attached. The purpose of 
this document is to promote discussion of the issues raised by participants regarding development of 
the 1992 National Adult Literacy Survey; and to gather input on such key topics as: reactions to and 
use of the 1992 survey, interpretation of the results, sub-population study and analysis, and the 
design and dissemination of findings. 

Reactions to and Use of the 1992 National Adult Literacy Survey 

Use and Benefit of the 1992 Survey 

Both the National and the State Adult Literacy Surveys have been useful to participants. 
Despite the fact that the Federal government can do synthetic estimates, state participants indicated 
that the investment in the state surveys was worthwhile. In New York, for example, the state survey 
was used as a needs assessment and to argue for education and welfare reform. In Illinois, where 
multiple agencies participated in the survey and expanded the questions, the state survey provided 
useful information not available from the synthetic estimates. It was also used as a needs assessment 
for developing educational services. Other uses of national survey data included the following: 

❖ At the Public Broadcasting Service results were used as justification for a national 
project to fund distance learning for adults. 

Information was used in speeches focusing on literacy issues. 

❖ The National Education Goals Panel set Level 3 as a literacy goal and called for another 
literacy assessment to see how quickly the population is reaching that goal. 

❖ Results were compared with other work done in the literacy field. However, at the local 
level it really was a “one day story” and reporters need a local angle to take it beyond 
that. 

❖ Information from the literacy survey was used as a marketing tool to encourage 
companies to invest in product development. Simon and Schuster, for example, invested 
in developing the Test of Adult Literacy Skills (TALS). 

♦t* At the Federal level, at the Department of Education, the 1992 survey data are used for 
all policy initiatives, as background information for Congressional testimony, and to gain 
support for special studies (e.g., welfare analysis, corrections, elderly population). 
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The 1992 survey has been used to influence policy and as a vehicle for program 
improvement. At least 12 states requested state sample supplements so that the information could be 
used at the local level. 

The literacy assessments would be more beneficial if characteristics of individuals within 
each level were described. Reports currently only talk about the tasks within each level (e.g., the 
items they had a high probability of getting right or wrong.) The reports need to describe the 
individuals at each level and not simply what they can or cannot do. 

Making the next literacy assessment 

Several recommendations were made to make the NAAL more meaningful to the general 
public. They include the following. 

❖ Link the Literacy levels to state measures. The general public does not understand 
what the levels actually mean. Texas, for example, wants to know how many people can 
read, write, add, and subtract but they do not understand what levels 3, 4, and 5 mean. 
The commissioner of Education in Texas is convening a new panel after two years to 
look at implementing new measures that will give the information they need to plan for 
instruction. . 

❖ Go beyond numerical scales. Use of a numerical scale allows you to assess what 
respondents can or cannot do based upon where respondents stand on the scale. One 
participant indicated that the public is confused by what the scale means. She suggested 
we move beyond a psychometric scale and see how the survey results relate to the job 
requirements of the workforce. Another participant suggested we could drop the 
numerical scale and talk about the richness of the information and what literacy is 
“enough” or “not enough”. However, the concept of “enough” is complex and requires a 
more targeted sense of adults’ goals in life and how literacy relates to those goals. For 
example, with X level of literacy, individuals would be eligible for X level of jobs in this 
country. Jobs, however, require both literacy skills and credentials. The question 
becomes how much literacy does an individual need who does not have a HS diploma. 
One participant suggested looking at the National Institute for Literacy's Equipped For 
the Future (EFF) project because it provides a rich source of data on the kinds of skills 
and knowledge adults need as workers, citizens, and parents. 

♦> Link survey results to research and practice communities. Conduct qualitative 
studies of learners at different levels on the scale to identify what they can or cannot do. 
TLMSS, for example, included qualitative supplementary studies at the 8 th grade level. 

❖ Report on literacy levels of diverse populations. Provide more descriptive information 
regarding the literacy levels of diverse populations (e.g., welfare, employed, unemployed, 
incarcerated, health). It is important to report out information that will help 
administrators manage their programs better and that meets the needs of diverse 
populations. 

❖ Characterize literacy skills that show what people can do rather than what they 
cannot do. Too often the focus is on what the population cannot do. 
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Interpretation of the next literacy assessment 

Linking the next literacy assessment to Other Surveys 

Linkages to several other surveys were discussed as well as problems with linkages. The 
following linkages were mentioned. 

❖ Linkage to NAEP’s 2002 reading assessment of 12 ,h grade students. This linkage was 
not made in 1992 as the structure of the NAEP blocks was different from that of the 1992 
Assessment. Participants thought we need to consider this linkage as an option. 

❖ Using literacy assessment data with other NCES surveys. For example, results from 
different surveys could identify the number of 4 lh graders who cannot read and who grow 
up and join the adult education target population. It is important to link sets of numbers 
into a coherent lifelong learning profile. For example, one could follow 4 th graders to 
adulthood or test 4 lh graders and their parents. 

❖ Linkage to international studies of adult literacy. Coordinate between the next literacy 
assessment and the U.S. component of the international survey of adult literacy. This 
linkage is critical but the process must be examined. Methodologies used in the 1992 
survey were different from those used in the 1994 U.S. component of the international 
study. Therefore NCES could not place the scores on the same scale. One difference 
was that the U.S. did not use incentive payments, which resulted in different response 
rates. There were also some operational problems. The data collection in 1994 missed 
everyone in colleges or universities so those figures were not comparable. The fact that 
there will not be an international survey in the same year as the next literacy assessment 
does not present a problem as there is not very much change from one year to the next. 

Response Probability Conventions 

Different assessments use different proficiency levels to report what respondents cannot do. 
Therefore we cannot compare how people are doing on the same items. The question raised was 
should there be a standard across different assessments so that everyone uses the same convention? 
This question was not answered at the strategy session. 

Use of Existing Subscales 

The next literacy assessment would require respondents to demonstrate their ability to use 
information contained in the printed material. The assumption made was that the same 3 scales 
would be used: prose, document, and quantitative literacy using a NAEP-like matrix sample of 
literacy tasks. One participant indicated that it was worth having subscales only if the scales were 
very distinctive and did not overlap as they did in the 1992 survey. Another participant indicated 
that the K-12 system does not distinguish between prose, document and quantitative literacy and that 
these scales were not resonant with reading, writing and arithmetic. Another said that these 
subscales were not useful at the state level because the state identifies different levels and, therefore, 
they found it difficult to share information from the 1992 survey with the press. 
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Composite Score 

Participants suggested using a composite score for prose and document literacy but keeping 
the quantitative scores separate. The synthetic estimates used a composite literacy scale. 

Other Types of Scales 

Several other types of scales were suggested including the following. 

Numeracy Scale. Numeracy was proposed to replace quantitative literacy. Differences 
between the two were articulated, such as: (1) numeracy includes quantitative literacy but is broader 
than quantitative literacy, (2) numeracy information is not always embedded in the text, (3) 
numeracy looks at a more authentic use of numbers rather than numbers for numbers’ sake, and (4) 
the quantitative scale uses a reading scale rather than a true quantitative scale, which is what 
numeracy would use. The question arose as to what happens to the trend measurement if one scale is 
changed. There are contextual effects that must be taken into account if one scale is removed. 

Writing Scale. Participants indicated that writing skills should be assessed. The K-12 
system is looking at writing assessments and if adult education wants to link with K-12 it would need 
a writing assessment. The writing assessment would identify those respondents who have OK 
reading skills but cannot write at all. It is possible to measure writing but it requires more items and 
it also would be difficult to assess writing from the reading questions. Writing assessments are 
significantly different from reading assessments and are very expensive. Several factors affect cost: 
(1) Experience from other surveys show the cost is related to survey administration time, not in 
developing the items. (2) Raters would need to be trained as the test items would not be the same as 
school writing responses. (3) The sample size would need to be increased to include writing. We 
may need a separate sample for the writing assessment. Levels 1 and 2 may not be able to respond 
on the writing assessment but respondents at level 3 may have differential outcomes. It was 
suggested that NCES look at the cost benefit of obtaining information through writing assessments. 

Persuasion vs. Evidence. A subscale to be considered in the more distant future was the 
ability to distinguish the difference between persuasive and factual (evidence) writing. 

Types of Items 

Several types of items could be included on the survey that would be helpful in planning for 
adult education programs. Among these items were health, employability, technology, and oral 
communication. Health, which could be assessed through document literacy, becomes increasingly 
important because of the need to read labels of non-prescription drugs. Oral communication was 
included in the 1985 survey but was dropped in 1992 because of the expense. Technology skills 
could be assessed through simple computer tasks. 
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Sub-population Study and Analysis 

Several issues were discussed regarding the analysis of sub-populations for the next literacy 
assessment. These were related to measuring the prevalence of populations with low literacy skills, 
including the homeless population in the survey, oversampling populations with special policy 
interest, and sample size. 

Prevalence 

Participants indicated a need to prioritize what we hope to get from survey. The question 
was raised as to whether to focus on special studies on the population at the low or high end of the 
scale or both. The level of education of the household could be used as part of the screening 
interview if the focus is the population at the lower end. 

The more important question regarding prevalence is not what is the prevalence, but how is 
prevalence distributed. To answer this question requires an examination of sub-populations which in 
turn requires a larger sample size. 

Homeless Population 

Another assumption stated in the overview was that the next literacy assessment will be a 
national, geographically based survey including all 50 states and DC but would not include: outlying 
territories or the homeless population who live in group quarters, institutions, or homes for the aged. 
The sample size would be smaller than the 15,000 target in 1992. Participants believed 
homelessness is a key policy issue and that it would be a mistake to exclude this population, as 
literacy is a key solution to the problem of the homeless. 

Welfare Population 

NAEP is interested in welfare history, not only geography and ethnicity. Background 
information in the survey should obtain history on the welfare population. 

Oversampling Groups with Special Policy Issues 

The groups identified in the project overview were rural adults. Native Americans, Blacks, 
and Hispanics. Participants indicated that there were too few rural adults in the 1992 survey to 
analyze literacy in the countryside. Native Americans represent the rural population as well. We 
could design the survey to make a separate estimate for that population. There was some discussion 
over whether it was necessary to oversample people in areas in which Blacks and Hispanics are 
concentrated. The high ability Blacks and Hispanics do not always live in those areas so it is 
possible we could end up with an undersample of that population. Therefore, it may not be desirable 
to oversample in these areas. 

Literacy Assessment in Spanish 

Politically, it is not possible to conduct the literacy assessment in Spanish. Participants 
generally believed that this option should not be considered. However, recommendations for 
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funding literacy assessments in Spanish were offered including obtaining support from private 
companies who have businesses in Mexico and who would be interested in the results. What may be 
important is what is the level of education in the native language. This is a factor to consider when 
setting up programs. 

Level 1 has too much in it and we need to segment out part of the Hispanic population. We 
were unable to obtain language acquisition on only about 6 or 7 percent of the Hispanic population. 
We need to sort out the non-English speaking population so that Level 1 will be less of a 
hodgepodge. 

Sample Size 

In the 1992 survey a larger sample size was required to make trend comparisons with earlier 
surveys. The sample size will be reduced in the 2002 survey. Participants were concerned about 
having a smaller sample size if we are concerned about analyzing sub-populations and recommended 
that it be increased. 

A population that could be added to the sample were parents in NCES’s Early Childhood 
Longitudinal Study to study the intergenerational transmission of literacy. The level of parental 
skills is a policy issue and should be examined and that the issue of parents’ skills changing has not 
been looked at. One participant stated that if young adults are participating in programs, we can 
assume their skill levels have changed. 

Designing and Disseminating Findings from the Next Literacy Assessment 

Participants discussed goals for the new national assessment of adult literacy, the need to 
enhance the background history and areas where potential papers may be useful for making final 
decisions about the assessment. These issues are described briefly. 

Identification of the Goals of the NAAL 

The goals articulated by participants were to: 

♦♦♦ Provide programmatically useful information for adults who desire additional education 
and skills development. 

❖ Provide more in-depth background information to allow for more in-depth analysis of 
data. 

❖ Provide data that is program relevant to the states. One recommendation for increasing 
sample size and obtaining valid state estimates without being too costly for states is to 
combine states that share similar characteristics and build in ways of reporting 
information that are program relevant. 

❖ Provide information for policy decisions (e.g., lifelong learning, homeless population). 
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❖ Provide a clear connection between literacy skills and employability. Employers are 
more concerned about behavioral skills than literacy skills. Identify those skills 
necessary for stable employment, not simply entry-level employment. 

*♦* Provide a connection between the school age learning disabled population and what 
happens to them in terms of adult literacy skills. Information collected through self- 
reports is not reliable. The International Adult Literacy Survey conducted a follow-up 
study on individuals who identified themselves as disabled, however through an 
administrative error (the skip pattern was not followed) the follow-up survey could not be 
used. There may be a possibility of crosswalking the next literacy assessment with the 
longitudinal survey conducted in Connecticut by the National Institute for Child Health 
and Human Development. 

*1* Provide information for managers to use to determine activities on the local level such as 
professional development. 

Interview Methodology 

An assumption stated in the project overview was that a household personal interview 
method is required to test adults ages 16 and over who are not in school. One participant suggested 
conducting telephone interviews, however there was disagreement on whether telephone interviews 
are a viable means for identifying whether people can read, write, and comprehend. Further research 
is needed before it is known whether telephone interviews are possible. 

Incentive Payments 

The assumption stated was that respondents will be offered a $20 incentive payment to 
compensate for the psychological burden of taking a test but there will be difficulty in convincing 
OMB to approve the incentive. Participants indicated that incentive payments raise response rates of 
individuals with low literacy. Use of incentive payments should be part of the field test. 

Background History 

Participants indicated the need for enhanced background information on survey participants 
in a number of areas. NCES, aware of this need, assumed that the background questionnaire should 
be lengthened from 22 to 35 minutes to cover the same major topics as before: language background, 
educational experiences, political and social participation, labor force participation, literacy 
activities, demographics. One participant indicated that the social and political participation 
questions on the 1992 survey were not useful. 



Participants indicated the need to enhance background information in the following areas to 
allow for more in-depth analysis: 



❖ Provide information on the characteristics of the unemployed, under-employed and those 
out of the workforce. This may be problematic as this population moves in and out of the 
workforce and would require a large sample for the churn pattern. 
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❖ Provide more information on educational experiences. The current survey provides 
insufficient information on low level education respondents because the survey was 
meant for a broader population. Lifelong learning is an important policy issue and the 
data on individuals who leave high schools and move on to other areas of education and 
training needs to be examined. At NCES there is no unit responsible for gathering 
information on lifelong learning. The next literacy assessment could begin the process of 
gathering such information. OECD is looking at continuing education. Stats Canada has 
a Recurring Adult Education Survey. Collection of education data (e.g., HS completion) 
will allow for comparisons with data from the Census or other NCES surveys. More 
substantive background information on education may be able to be related to tests that 
are being used to assess adult education learners (TABE, ABLE). Also include the level 
of education in the native language because this information affects how adult education 
programs are set up. 

Provide more information relative to participation in basic skills training programs, etc. 
This information was lacking in 1992. The problem is that participation of individuals in 
training programs is so small, that it increases the difficulty of capturing them in a 
sample. The smaller the sample size, the greater the risk in underrepresenting them. A 
solution is to oversample the population with lower education levels to improve the 
chances of incorporating people in training programs, but it will still be small. 

*1* Provide information on the numbers of respondents who have literacy needs. Such 
questions could include: Do you think you have a literacy problem? Would you do 
something about it if you had an option? And what would you do? Analysis of such data 
would provide a richer answer to the question. How many of these individuals are there? 

•J* Provide information on language use and facility in other languages. This information 
was collected in the 1992 survey but was not disseminated. 

i* Provide information on citizenship issues (e.g., How closely do respondents follow 
public affairs? Where do they get their information?) 

❖ Participants agreed that the descriptive information obtained through the survey is 
critical. One participant indicated that with a finite amount of time and resources to 
conduct the survey we may need to prioritize the type of information collected and make 
it less generic. Priority areas could include employment, lifelong learning, and needs 
assessment. 

Improvement in Survey Methods 

The background survey will be pre-tested through the use of cognitive laboratory procedures. 
Data collection may be improved through computer assisted personal interviewing. Computer 
assisted personal interviewing helps when there are large skip patterns in the questionnaire. 

Collaboration with Other Agencies 

Participants recognized the costs involved in administering and reporting a large survey like 
the National Assessment of Adult Literacy. A focus on various sub-populations requires an 
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increased sample size and therefore additional funds. Participants recommended that NCES 
collaborate with other agencies concerned with the same population to obtain additional funding for 
the survey. In Canada, for example, funds come from multiple sources (e.g., the Senior Secretariat, 
Human Resources Development, the Census Bureau, the Department of Industry, and the provinces). 
NCES should consider collaboration with the Departments of Justice, Commerce, Labor, Health and 
Human Services and well as other offices within Education (e.g., OVAE) to leverage resources. It is 
important to make other agencies and offices see adult literacy as an interrelated problem and how 
they could benefit from the information gathered through the next literacy assessment. A major role 
for NCES in the collaboration would be to provide advice on the design of the survey. Two 
examples of collaborative efforts were provided. Stats Canada convinced other agencies that this 
information is beneficial to them and collaborates with such entities as the Senior Secretariat, 

Human Resources, Census Bureau, the provinces. Illinois collaborated with other agencies in 
funding the SALS, identifying questions to include on the survey, and reporting on issues. Each 
agency had a separate contract with ETS, which reduced the administrative burden on the 
Department of Education 

To enhance the collaborative effort at the Federal level and make the survey useful there 
needs to be upfront coordination and planning. A NAAL National Advisory Committee with 
Assistant Secretary representation from each of the agencies should serve as the policy group for the 
study. The collaborative process provides: 

♦> A means for leveraging resources; 

*•* Agencies with the opportunity to interpret results from their own policy perspectives. 
NCES cannot report on policy implications of survey results survey but through a 
cooperative agreement, other agencies can use their funds to analyze and report data and 
discuss the policy implications that are relevant to them. Other entities could conduct a 
secondary analysis and prepare reports without NCES’ name on the report. 

❖ Greater attention to the issues. 

❖ A communication link between K-12 and adu|t literacy. Investment in adult literacy is 
important because adults are parents and citizens. 

♦♦♦ A model for states to follow in implementing the state literacy assessments, either 
independently or through sample supplements to the national survey. 

Potential Discussion Papers 

Several topics raised during the strategy session require further investigation to help NCES 
make final decisions regarding the new assessment of adult learning. They include the following. 



❖ How to make interagency collaboration work. Participants were enthusiastic about 
collaborating with other agencies in funding, analyzing and disseminating results. 
Further investigation is required to determine if this is feasible. 
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❖ Cost-benefit analysis of adding a writing assessment. Participants were in agreement 
that writing is a very important aspect of literacy and should be added to the survey. 
However it is quite costly and NCES needs to determine whether it is worth the cost. 

❖ Special studies of cohorts within adult education programs. The question raised was 
should there be other studies that look at specific populations such as the elderly or 
immigrants. Canada conducts focus surveys as a follow-up to the main survey. 

❖ Focus of the Survey. The question raised was whether the survey should focus more on 
adults with lower level skills than the broader population. The answer requires further 
investigation. 

❖ Interview Methodology. The last survey was conducted through personal interviews. 
Further research needs to be conducted to determine whether telephone interviews are a 
viable alternative. 

Usefulness of a New Survey 

Participants agreed that a new survey would be quite useful. Comments included: 

❖ The 1 992 data is old and is getting more difficult to “sell”. 

❖ The new survey would provide useful trend data. 

❖ Media education writers want a better understanding of competencies as the workforce 
changes. 

❖ More in-depth background information would be useful for analysis. 
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Strategy Session: February 2, 1998 

The second strategy session for the purpose of developing a National Assessment of Adult 
Literacy (NAAL) to be conducted in 2002 was held on February 2, 1998 at the offices of Pelavin 
Research Center in Washington, DC. A list of attendees is included in Attachment A. 

The session began with an overview, by Andrew Kolstad, of project goals, assumptions, and 
schedule, as well as unsolved problems. A summary of this overview is attached. The purpose of 
this document is to promote discussion of the issues raised by participants regarding development of 
the next literacy assessment; and to gather input on such key topics as: reactions to and use and 
interpretation of the 1992 survey data, sub-population study and analysis, the design and 
dissemination of findings, and the State Adult Literacy Assessment. 

Reactions to and Use of the 1992 National Adult Literacy Survey 

Participants raised two questions regarding the use of the 1992 survey: (1) Who are the 
target audiences? and (2) How do these audiences want to use information gathered from the survey? 
Among the target audiences identified were Congress, the media, the National Institute for Literacy 
(NIFL), and state directors of adult education. Several uses of the 1992 data were mentioned. 
Members of Congress are interested in knowing how their population’s literacy compares in relation 
to literacy in other states. NIFL uses the information to respond to calls from the media asking how 
many people are illiterate. Data from the 1992 survey were used by states to put together talking 
points that helped to gain additional funds at the state level and at least partially at the national level. 



Problems in Using Information from the 1992 survey 

Several problems were cited regarding the use of the information. These difficulties are 
identified below. 

❖ There is a need to present the information on a continuum in a more user-friendly way to 
describe who the population is and where they fall on the continuum. 

❖ People did not understand the meaning of the five literacy levels. To make it more 
meaningful, one participant equated the levels to high school graduation and beyond and 
identified the skills people should have at these levels. This correlation may have created 
incorrect information, but it created a basis for discussion. Participants indicated a need 
to connect the levels to something people could understand (e.g., grade levels or a 
diploma). 

❖ A definition of literacy is lacking. It is difficult to say people have “enough” literacy 
when literacy has not been defined. 

A lack of congruence exists between an individual’s self perception of literacy skills and 
literacy as measured by the NALS. The 1992 survey showed that 66-75 percent of the 
individuals in Level 1 thought they could read well. From an adult literacy perspective it 
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is important to see how people perceive their own abilities. Individuals are more likely to 
seek help in improving literacy skills if they perceive a problem rather than if the results 
of a Federal survey say there is a problem. To obtain a more accurate self-assessment, 
one recommendation was to ask people how well they thought they could read after 
completing the test items. The answer may be different after the survey than before. 
Another suggestion was to release old items from earlier surveys and allow people to 
assess themselves. A compendium was prepared for NIFL that contains sample items 
from prior surveys. 



♦♦♦ Data in the reports are seven years old and are not useful now. Society and state 
demographics change and it is difficult to make projections down the road. Linear 
models do not work. Decisions made now, for example, would not be relevant 10 years 
from now. What states need is a feedback system that is ongoing and continuous. 

Making the next literacy assessment more meaningful 

Several suggestions were offered to enhance the use of the data from the 1 992 survey. They 
included the following. 

❖ Cross-reference the next national literacy assessment data with other adult literacy 
assessments used in the states such as the TABE or ABLE. A broad national sample 
using a quick checklist with vocabulary, and then correlations with the state assessments, 
would begin to help people crosswalk the data. Participants noted that the National 
Center for Adult Learning and Literacy is looking at linkages between other assessments 
and the national adult literacy assessments through a number of local studies in 
Massachusetts. 



❖ Crosswalk the data with the literacy levels that are being standardized in the National 
Reporting System. This is what states will be required to report on in the future and for 
which they will be held accountable. 

❖ Learn from the Canadians how they have taken the information and mobilized the media 
around the issues. 

❖ Consider the timing of the release of reports. If Congress is the target audience, one 
participant suggested that the release of data from the next adult literacy assessment 
should be timed to coincide with the cycle during which Congress reauthorizes adult 
education. It is only at that time that the report will attract Congressional attention. 

❖ Generate interest in the next adult literacy assessment, through an Internet web site, 
provide information about the items and what is happening with the items. The old items 
from the 1985 and 1992 assessments could be placed on the literacy web site. 

*1* Consider a norm-referenced assessment that shows where people stand in relationship to 
one another, rather than reporting an absolute standard. 



Link the next literacy assessment to other studies. It will be more useful if it is tied into 
the literacy population estimates gathered from other studies. 
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❖ Find a mechanism to tie the next literacy assessment back to the demographic and 
dynamic changes in society. 

❖ Base the next literacy assessment on some sound validated theory about literacy. It is 
difficult to say enough about what is being assessed because these are complex 
information processing tasks and the difficulties are attributed to overloaded working 
memories, not literacy. The problem is they are all confounded and it is difficult to 
determine why people perform poorly. We don’t know if they are ignorant of the subject 
matter or if they are being asked to hold multiple things in their memory while 
performing another task. One recommendation was to assess people’s knowledge. To 
assess their literacy abilities, assess the knowledge that is most likely derived from 
literary sources. A simple vocabulary test, well designed, provides almost all the 
variance the NALS tasks provided and almost all the same validity as the literacy tests. 

Aspects of the 1992 survey results that Were Easy to Present 

Participants indicated that several charts in the 1992 adult literacy report were useful. Of 
particular importance was the chart that demonstrated the link between parents’ education and their 
children’s level of achievement. (See Figure 1.4 attached. Average Literacy Proficiencies by Level 
of Education Attained by Adults and Their Parents). This chart was useful in the state to help shift 
the perspective of adult education from remediation to prevention of illiteracy among the next 
generation. It served “double duty dollars” by showing that you are helping children when you are 
helping adults. 

The other useful charts were related to literacy and economic levels. (See Figure 2.9, 

Median Weekly Wages by Literacy Level, and Figure 2.10, Percentage of Adults in Certain 
Occupational Categories by Literacy Level). These charts demonstrated the link between earnings 
and occupations and literacy levels. 



Interpretation of the next literacy assessment 



Use of Existing Scales 

Participants indicated that three scales were difficult to use. When reporting information to 
the public they could only interpret one scale because it would be confusing to work from all three 
scales. They also noted that the scales are highly correlated. The difference noted in the scales is 
most clear when ethnic groups are examined. Where differences appear (e.g., Asians score better on 
the quantitative scale. Blacks score less well on this scale) the differences should be reported. 

Participants also suggested changing the scales to math and verbal as these are the terms with 
which people are most familiar. 

Composite Score 



Participants noted that the prose and document scales are not clearly differentiated and 
recommended a composite score, while keeping numeracy separate. 
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Other Types of Scales 

While several other types of scales were recommended, one participant cautioned that 
additional scales may make reporting more complicated. The reporting could be simplified, 
however, if the initial report were on literacy and subsequent reports focused on the following scales. 

Writing Scale. Participants suggested assessing writing skills through a holistic writing 
sample such as that used by the GED. Writing scales were used in K-12 arena and writing skills 
improved. 

Oracy Scale. One participant suggested including an oracy scale. Individuals with highly 
developed oral skills will, once they develop decoding skills, also develop high literacy skills. It is 
important to determine if people are as effective and efficient in written language as they are in the 
spoken language. It is possible to build parallel items that will allow you to say that people can read 
as well as they perform orally. 

Technology Scale. Participants agreed that basic computer literacy is a basic skill that 
should be assessed. Simple questions could be added to the background survey that would indicate 
information about whether respondents know how computers function. One participant suggested a 
telephone survey to gather background information on computer literacy using simple questions such 
as Is there a computer in the home? If there is a computer, do all family members use it? 

Link the NAAL to Other Surveys 

Participants recommended linking the NAAL to other surveys to provide important reference 
points for comparison and to make the findings more meaningful to the public. One 
recommendation was to link it to the International Adult Literacy Survey (LALS) to compare how 
adults in the U.S. compare internationally. The other recommendation was to link the NAAL to the 
National Assessment of Educational Progress (NAEP) reading scales. Such a linkage would show 
how many adults read at the grade levels used by NAEP and would demonstrate developmentally 
how we look in the U.S. Such data would report information that is more familiar to people. 



Sub-population Study and Analysis 

Several issues were discussed regarding the analysis of sub-populations for the NAAL. 

❖ Native American population. Our cultural perceptions of literacy are different from the 
perceptions of Native Americans, and these perceptions affect literacy issues. For 
example, Native Americans’ verbal heritage may affect their written emphasis and 
literacy practices. If the Native American population is to be assessed, it is important to 
focus both on those living on reservations and those living in metropolitan areas in order 
to obtain the full picture. 

❖ Welfare population. Some participants indicated that while welfare is a “hot” issue 
currently, it may not be a major issue seven years from now. Perhaps the focus should be 
on those welfare clients who are still in the system and cannot make it off the welfare 
rolls. 
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i i ► Immigrant population. Participants thought immigration will continue to be a big issue 
in the future. They indicated the importance of looking at intergenerational transfer and 
how it differs by ethnicity. One recommendation was to look at data from the National 
Center for Family Literacy and from Even Start. The ratings used by these groups should 
provide useful information regarding intergenerational transfer. 

❖ Urban population. Data on literacy in urban areas would be useful to mayors and urban 
coalitions such as the Council for Great City Schools. 

❖ Individuals with disabilities. The 1992 NALS gathered data on visual disabilities but 
the deaf population was excluded from the survey. Respondents were asked to report 
whether they had other disabilities. Participants suggested a separate survey be 
administered to the learning disabled population as it is important to understand how 
much learning disabilities contribute to illiteracy. They recommended a joint study with 
the National Institutes for Health. 

❖ Population between 16 and 18 years of age. The adult education system is currently 
facing challenges from this age group who are impacting on the traditional adult 
education population. This group may have to be oversampled in a survey. 

♦> Elderly population. Congress requires this population be included in the survey. 

Assessing the elderly population will show us the dynamics the U.S. will face as a nation 
as baby boomers move into old age. Literacy levels drop off after age 65. Studies have 
shown that an individual’s processing skills continue to grow through adulthood and then 
drop off after ages 40-45. However, individual’s content or declarative knowledge 
continues to grow if these adults remain active. 

❖ Literacy assessment in Spanish. The Hispanic population argues that they should be 
assessed in Spanish. They believe it is denigrating to be placed in the illiterate category 
because of their lack of English. The question raised during the discussion was Can we 
get a true picture of how people succeed in the U. S. in one language only? We have no 
information on how well individuals function in society who are non-English speaking. 



Designing and Disseminating Findings from the 
New Assessment of Adult Literacy 

Participants discussed goals for the new national assessment of adult literacy, ways for 
reporting data that would make it more meaningful, and areas where potential papers may be useful 
for making final decisions about the assessment. Two questions they thought would be important to 
answer through the NAAL data were (1) Can I compete worldwide? and (2) Can I compete with my 
fellow citizens? Answering these questions requires using norm- referenced criteria. 

Identification of the Goals of the next literacy assessment 

Participants articulated several goals. These included: 
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❖ Provide the data to make policy decisions regarding such issues as justification for 
funding for adult education programs and teachers. 

❖ Provide the data to make instructional decisions to help adult educators do a better job. 

❖ Keep the issue in front of policy makers and high level people in Congress. 

❖ Provide a national status report and allow the public to see the literacy trends. 

❖ Allow states to make comparisons between literacy trends in their states and the nation. 

❖ Allow us to make comparisons with the rest of the world. If we, as a nation, want to 
answer the question of whether we have skills to be successful in a high performing 
economy, we will need to look at the whole range of literacy and not focus only on the 
low levels of literacy; an area of concern for those in the adult education field. Levels 4 
and 5 are equally as important. It is important to identify the minimum level of 
proficiency to be a leader in the world. 

In their discussion of goals, participants returned to the question of who is the audience for 
this survey. For some it was state legislators, for others it was Congress. If Congress is the 
audience, participants indicated the need to get states involved. Unfortunately, state budgets of small 
states are insufficient to fund state analyses. 

Recommendations for Reporting from next literacy assessment 

Several suggestions were offered for reporting the data. They included the following. 

❖ Keep the report short and easily digestible for members of Congress. Congress is 
interested in the prevalence of populations with low literacy skills and the trends in 
literacy. They want to know (1) whether we have made a difference since the last survey, 
(2) how many people in their state need help, and (3) how the literacy skills of people in 
their state compare with the literacy skills of people in other states. The most important 
piece is the Executive Summary, which should describe the scope of the literacy problem 
and the conclusions that can be drawn from the data. One participant also suggested 
conducting focus groups on Capitol Hill to gather information from Congressional staff 
on what information would be most important for them. 

❖ Identify the cultural barriers to literacy. Knowledge of the cultural barriers across 
cultures may be a way of unlocking the keys to literacy. 

❖ Provide data on the intergenerational component as well as on gender. 

❖ Compare the condition of adult literacy to results from postsecondary assessments. 

❖ Use a norm-referenced scale rather than a criterion referenced scale, particularly if the 
goal is to influence Congress. 

❖ Report the number of people at Level 1 who are on a waiting list for an adult education 
program. This type of data will gain more attention for adult education. 
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Background History 

Participants indicated that the background information should be consistent and believable. 
In the 1992 survey there was a big disparity between respondents’ self perception of their literacy 
skills and the results of the survey. Such disparity tended to negate the survey findings. 



Participants indicated the need for more descriptive data about the population. Information 
such as the following would allow for more in-depth analyses: 

❖ Provide information on the reading habits of the population. For example, an earlier 
survey found that if respondents read more than 6 times per week, holding age, education 
and ethnicity constant there were large differences in income, occupational status and 
political activity. 

❖ Provide information on continuing education experiences, including training at work. 
Participants believed that recreational education courses should be recorded separately. 

❖ Provide information on personal financial competence. The concern was raised, 
however, that such information was intrusive and would require OMB justification. A 
general question (e.g.. Do you have the necessary skills to handle your financial affairs?) 
may suffice in gathering the information. 

Materials or Training on the Use of the data from next literacy assessment 

Participants provided several recommendations that would make the forthcoming data and 
reports more user friendly including the following: 

❖ Provide an Executive Summary that is substantive. 

❖ Have the scale designed in such a way as to make it easily explainable. 

*1* Keep the charts in the reports simple to allow them to be used as talking points. 

❖ Provide a one page overview for policy makers. 

♦♦♦ Provide information that is easily visualized. For example, chart national information 
next to state information or place national information on the front page and state 
information on the back of the page. 

❖ Provide a simplistic design of the survey for states to use. A survey with fewer questions 
would be less costly. 

Potential Discussion Papers 

Several topics raised during the strategy session require further investigation to help NCES 
make final decisions regarding the new assessment of adult learning. They include the following: 



Standardized response probability conventions. Different surveys use different levels 
of probability. Currently NALS and the IALS use an 80% response probability 
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'i 

convention, CASAS, uses 50% and TIMSS and NAEP use 65% for item mapping. 

NCES needs to determine which standard to use. 

❖ Feasibility of including other populations within the survey. Sub-populations for 
study include the learning disabled, the homeless, and the second language populations. 

❖ Ability to improve reading skills. Data is needed from adult basic education programs 
that show that peoples’ literacy skills can improve through instruction. Such information 
will counter the perception that people cannot improve. 



State Adult Literacy Surveys 



Participants suggested that NCES ask Congress if they need state by state data or whether 
synthetic analyses such as those done with the 1992 data would suffice. For the new survey the 
synthetic analysis would be based on data from the next Census. 
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Strategy Session: February 10, 1998 

The third strategy session for the purpose of developing a National Assessment of Adult 
Literacy (NAAL) to be conducted in 2002 was held on February 10, 1998 at the offices of Pelavin 
Research Center in Washington, DC. A list of attendees is included in Attachment A. 

The session began with an overview, by Andrew Kolstad, of project goals, assumptions, and 
schedule, as well as unsolved problems. A summary of this overview is attached. The purpose of 
this document is to promote discussion of the issues raised by participants regarding development of 
the next Adult Literacy assessment; and to gather input on such key topics as: reactions to, use and 
interpretation of the 1992 National Adult Literacy Survey, sub-population study and analysis, the 
design and dissemination of findings, and the State Adult Literacy Assessments. 

Reactions to and Use of the 1992 National Adult Literacy Survey 

Participants raised several difficulties in using the 1992 results and offered suggestions for 
improving the usefulness of the next assessment. 

Problems in Using Information from NALS 

Among the problems encountered were the following: 

❖ The use of three different scales and five levels of literacy made it difficult to report the 
information to the public and to legislators who relate to grade levels and K-12 
standards. The literacy levels, particularly at the middle and high end, meant different 
things to different people. 

❖ Literacy was not clearly defined. Some states presented literacy levels in a K-12 format 
to make it more meaningful to people. A public relations strategy was needed, in 
advance of the survey, which defined literacy and translated survey findings in a 
meaningful way. 

❖ The Limited English Proficient (LEP) population did not show up clearly in the reports. 
As a result, in New York State, for example, where there is a large LEP population, the 
scores were particularly low. The low level of literacy was equated with the need to fix 
schools. If the LEP population had been pulled out of the scale, the scores would have 
been higher. 

❖ The simplistic view of the scales masked issues of policy importance. 

❖ In one state, the Governor’s Office wanted the Department of Education to wordsmith 
the report to place the findings in a more favorable light. Policy implications got lost 
when this occurred. 

❖ Findings were reported out of context. There is a need to report data in a way in which 
people will understand. 
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❖ There was difficulty getting reports focusing on the workforce. While interesting 
information existed, charts were lacking to substantiate the information. 

❖ A lack of congruence existed between an individual’s self perception of literacy skills 
and literacy as measured by the 1 992 survey. The respondents in Levels 1 and 2 did not 
perceive themselves as having a literacy problem. They do not see themselves at risk 
with a changing labor market. Their skills are adequate for their work and home 
environments. 

Making the next literacy assessment More Meaningful 

Several suggestions were made to make the next literacy assessment more meaningful to 
legislators and the general public. 

❖ Develop a public relations strategy. Before and after the survey, develop a public 
relations strategy to translate the information for legislators and the general public. 

There is a need to inform the public better about what the data mean and the implications 
for policy. Follow the Canadian example and play a more active role in developing a 
public relation strategy. 

❖ Clarify what it is that we want to report. We may need two separate reports: (1) a 
“quick” and “dirty” snapshot for the general public, and (2) more in-depth analysis for 
people who have to work with the findings and make programmatic and policy decisions. 

❖ Educate the public that reading goes beyond the ability to decode words. Encourage 
people to think about reading as a continuum with the ability to decode words, synthesize 
information, and apply information. A better understanding of what reading is will help 
them better understand the results of the survey. 

♦> Report information within a broader context. Two examples were provided. (1) 

Look at workforce development efforts and relate information about the skills workers 
have and the skills they need to be proficient in the workforce at various levels (e.g., 
entry level, advanced). (2) Look at parenting skills and report information on the number 
of parents who lack skills to support their child’s educational progress and the skills that 
these individuals need to support their child’s education. 

❖ Involve political groups. Let political groups, rather than researchers, make the 
decision on the types of parenting, job-related, and entry-level skills necessary to 
function within these specific contexts. 

❖ Do not report information by grade levels. 






Developing the National Assessment of Adult Literacy: Recommendations from Stakeholders 



Interpretation of the next literacy assessment 

Other Types of Scales 

There is a notion that other skills (e.g., problem solving, teamwork, oral and written 
communication) relate back to literacy and that there is a relationship between literacy and the 
economic dimension. Analyses show that there is a labor market reward for literacy. 

It is possible to add other scales. However, time and money factors need to be taken into 
account. We need, therefore, to prioritize the skills we assess. Several other types of scales were 
recommended. 

Problem Solving and Team Work Scale. One participant related the difficulties in 
measuring these skills. In one state, a performance assessment was used but it was fraught with 
difficulties and removed. Another participant suggested that if listening and speaking could be 
assessed, it would inform the issue of problem solving. 

Writing Scale. The 1989 Canadian assessment had difficulty coming up with authentic 
writing tasks. There are pragmatic measurement problems in assessing adult writing. The GED has 
set a writing standard. However, looking at gradations below the standard is difficult. One 
participant suggested that a scale with writing standards be established, in advance, and that people 
could be placed on the scale at different cutpoints. However, it could be a political issue on how the 
cutpoints are set. 

Technology Scale. ETS uses a technology scale in the Test of English for Foreign Language 
Students that measures computer familiarity. It is a one page questionnaire, easily scanned, that 
provides information on the respondent’s access to computers (home or work environment), attitude 
about computers, facility with related technology, and ways in which a computer is actually used. 
The Life Skills project in Canada is planning to use this scale. This scale could be used to collect 
baseline data and then assess how peoples’ familiarity changes over time. While the information 
does not assess cognitive abilities, the findings could be used to change the way instruction in adult 
education is provided. The technology survey could be conducted over the telephone. State 
representatives thought this would be useful information. It would also have implications at the 
national level where there are several technology initiatives. 

Composite Score 

Participants did not see a need for composite scores. They were concerned that if only one 
scale were chosen, teachers would teach to that scale and that would be the only skill that would be 
discussed. They suggested one scale could be emphasized in relation to some issues and another 
scale emphasized in relation to other issues. For example, the document scale could be emphasized 
in relation to employability and the prose scale emphasized in relation to family issues. The focus 
would be on the scale most important for the specific context. 

Another reason for not developing composite scores is that different subgroups score 
differently on different scales. This data would not be shown on a composite scale. 
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Level 1 

One participant was concerned that Level 1 was too broad. It combined individuals who 
provided no response (about 5%) with those who could perform limited tasks. It was hard to identify 
the true non-literate population. 

Link the next literacy assessment to Other Surveys 

Participants did not believe it was necessary to link the NAAL to the National Assessment 
for Educational Progress (NAEP). The linkage would be more beneficial for the K-12 population 
than for the adult education population. The emphases of the two assessments are different. NAEP 
emphasizes academics while the next literacy assessment will emphasize application. Because of the 
lack of academic content, comparing the NAEP and the next literacy assessment will not further the 
discussion of whether there should be greater depth in content and arithmetic functions at the K-12 
level. One participant also indicated that the risk in comparing the next literacy assessment to the 
NAEP would be that the K-12 system would be seen at fault if the scores on next literacy assessment 
were low. 



Participants thought it would be more important to link the next literacy assessment to the 
Early Childhood Longitudinal Study. One state is trying to focus on adult learning outcomes and 
student progress at K-12. As one participant indicated, there is a need to “break the glass ceiling” 
between adult literacy and children’s progress. 



Not all state representatives thought it was necessary to link the next national literacy 
assessment to other state assessments of adults. 

Sub-population Study and Analysis 

Several populations were identified for potential further analysis. 

❖ Speakers of other languages. 




❖ Incarcerated individuals. There was some confusion over the prison vs. jail population 
in regards to the length of the average stay in prison and the sentenced vs. the non- 
sentenced population. Participants thought that it was more important to look at people 
in jails than in prison, because they felt that once the individual was in prison, you would 
have less impact on them. Those with a sentence spend six months to a year in jail. The 
non-sentenced cycle through more quickly. Sentencing needs to be defined. 

❖ Low income rather than welfare population. Participants believed that income level 
rather than welfare per se was the issue. Looking at welfare recipients increasingly will 
become more difficult with the new welfare legislation. However, we will continue to 
have a poor population that is not on welfare. 



Rural population. The rural population was considered important, however, there was 
some confusion about the definition of rural. 
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♦> Literacy assessment in Spanish. It is costly to do an assessment in two languages and 
we would need to establish priorities for sub-population studies. 

Designing and Disseminating Findings from the New Adult Literacy Survey 

Participants discussed goals for the new adult literacy survey and information that should be 
included in the background section. 

Identification of the Goals of the next literacy assessment 

Participants articulated several goals. They included: 

❖ Provide data to make decisions about the programs that are being provided for their 
populations; 

❖ Provide a contextual orientation for literacy by looking at workers’, parents’, and 
citizens’ needs; and 

❖ Provide data to make policy decisions including funding to support programs for limited 
English proficient populations. 

Background History 

Some participants indicated the need for enhancing the background survey and how the 
information is reported. In determining the length of the background survey, the next literacy 
assessment will need to balance the response vs. the resource burden. 

Participants recommended that background information be provided on: 

❖ Whether individuals participated in adult education programs, to help determine if 
parental participation impacts children’s learning; 

❖ Whether individuals were under correctional supervision; 

❖ How behavioral skills are used in daily life; and 

❖ Whether individuals participated in continuing education/training programs. 

Other Issues 

Participants raised the issue regarding the large amount of Pell Grant funds that support 
remedial instruction at the college level that are not part of the adult education system. 
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Potential Discussion Papers 

Two issues were raised for further investigation: 

❖ The relationship between parental participation in adult education programs and 
children’s educational progress; and 

❖ Accounting for speakers of foreign languages. 

State Adult Literacy Surveys 

Three of the four states participating in this strategy session participated in the 1992 State 
Adult Literacy Surveys. The state that did not conduct a supplemental survey indicated that data 
supplied as synthetic estimates, while broad, was sufficient. States that participated indicated that 
they did so for the following reasons: 

❖ To provide a different perspective from the census data for looking at literacy; 

❖ As a prerequisite for setting the state’s literacy agenda; and 

❖ To respond to the need from the provider network to structure learning activities. 

Funding of State Survey 

States varied in how they funded their state survey supplement. Two states supported it with 
Section 353 funds. Another state used a combination of Section 353 funds, and funds from JTPA 
and Education for Gainful Employment. In this state, the smallest portion of funds came from 
Section 353. The agencies that supported the survey helped to develop the questions. 

One state contracted with a university within their state to replicate the national survey, while 
the others participated in the broader, national survey. While it was initially assumed that 
contracting within the state would be less costly, the complexity of issues associated with the 
assessment resulted in increased costs and timelines that were overrun. In addition, there was no 
money left in the budget to sufficiently analyze the data that was collected. It was also recognized 
that if the assessment contractor tries to service all states, on a narrow timeframe and a fixed budget, 
it is not possible to tailor each report to the political needs of the states. 

Ways in Which the Survey Was Used 

The ways in which survey results were used varied with the states. In one state, internal 
affairs precluded the dissemination and use of information; the reports were held at the State 
Department of Education. In other states the survey was used in the following ways: 

❖ Findings were used to raise awareness of the literacy levels by regions within the state 
and between the state and its geographical region and the nation. 

❖ Monthly press releases are issued by topic area (e.g., workplace, family literacy) to keep 
the public informed about literacy issues. 
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♦> The findings were used as benchmarks and the state is building on these benchmarks to 
measure progress in improving literacy. 

❖ The information was taken to the State Board of Education and a Task Force for Adult 
Education was established. 

❖ Different agencies have used the information from time to time, including the Workforce 
Development Board. 

❖ Survey findings led to a discussion to develop an assessment framework for adult 
education and literacy. The framework parallels writing, speaking, and listening skills. 
There is a legislative mandate to have the assessment system in place by 1999. 

❖ Programs use survey findings to justify support for adult education. 

One of the difficulties faced by state agencies is that they did not know how to interpret the 
data they received. ETS provided short responses to specific questions but this was not sufficient to 
meet the needs of individual states. Real technical assistance is needed to help states interpret data. 
Participants indicated that policy issues (e.g., dissemination, data analysis) need to be considered 
early in the process. 

Participants indicated that the survey findings did not change operationally what they were 
already doing at the program level. 

Usefulness of a New Survey 

Some state participants indicated that another survey would be useful to measure trends over 
time. However, Section 353 funds may not be available to conduct another state survey. In one state 
it would take a grassroots and interdepartmental movement to gain endorsement for the survey. If 
there were financial incentives from the Federal level, it would boost advocating for the 
supplemental survey. Many states already rely on synthetic estimates and this may be sufficient for 
other states as well, despite the fact that synthetic estimates only provide information about the broad 
population. 

One state indicated that a new survey could help leverage funding for adult education 
programs serving non-English speaking adults in the city. Help is needed because the fiscal formula 
is biased against cities, precluding the ability to serve many non-English speaking adults. Foreign- 
bom residents who received a high school diploma in their native country do not count for aid either. 

One state indicated that they already had sufficient data about literacy needs from 
information presented in proposals to the state. They could use this information to make a case for 
adult education at the legislative level. 
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I. Project Goals 

NCES will conduct in 2002 a national sample survey of U.S. adults that assesses their 
literacy skills in a manner comparable to the 1992 National Adult Literacy Survey. NCES 
will publish a report on the initial findings of the survey, a report documenting the 
procedures used, and a data file for secondary analysts. 

A. Prevalence measurement 

Results will enable the identification of populations with low literacy skills and help 
literacy practitioners identify client populations and target literacy programs. Improved 
measures. 

B. Trend measurement 

Results will meet the need for monitoring trends in the status of adult literacy in America 
implied in the sixth national goal for education: that every adult become literate. 
Unchanged measure. 

C. Interval precedent 

The project will institutionalize at NCES a regular program of adult literacy assessments 
at one-decade intervals. 

II. Project Assumptions 

A. A household personal interview method is required to test adults ages 16 and over who 
are not in school. 

B. A national, geographically based survey, including all 50 states and DC, would collect 
background data through oral interviews, then provide printed material to which the 
adults would read and respond. 

1 . The national survey will not include the outlying territories. 

2. The national survey will not include those who are homeless or who live in group 
quarters, institutions, or homes for the aged. 

C. The federal government would provide an ongoing forum (periodic conferences) for any 
states that purchase a state supplemental sample, so that those states would be kept 
informed about the status of the project. Their input would be sought on decisions that 
might affect them. 

D. The assessment will measure at least two of the three scales previously used (prose 
literacy, document literacy, and quantitative literacy) using a NAEP-like matrix sample 
of literacy tasks. 

E. Those literacy tasks originally developed for the 1985 Young Adult Literacy Assessment 
will no longer be used, and new literacy tasks will be developed to replace them. 
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F. Respondents will be offered an incentive payment of $20 to compensate for the 
psychological burden of taking a test — a task that may be aversive for those with poor 
literacy skills. Obtaining OMB permission for this payment will not be routine. 

G. The associated background questionnaire will be extended from 22 minutes in the 1992 
survey to 35 minutes in length and will cover the same major topics as before: 

1 . language background, 

2. educational experiences, 

3. political and social participation, 

4. labor force participation, and 

5. literacy activities 

H. New developments in survey methods should improve quality: 

1. The background survey will be pretested with cognitive laboratory procedures 
including "think-aloud interviews" and linguistic coding of interview behavior to 
locate the sources of difficulties that respondents have with interview questions. 

2. Computer assisted personal interviewing may improve data acquisition. 

III. Possible Project Options 

A. State sample supplements to achieve sufficient size for state reports. 

B. Creation of state-level projections for states without supplements. 

C. Substituting "numeracy" for quantitative literacy, if successfully developed by the 
Canadians for the International Life Skills Survey. 

D. Another assessment of literacy in prison. 

E. A linkage to international studies of adult literacy. 

F. A survey supplement to test linkage to telephone method. 

G. Sufficient sample sizes for groups of special policy interest: 

1. Rural adults. 

2. Adults in welfare-to-work programs. 

3. Immigrants and non-native speakers of English. 

4. Native American Indians. 

5. - Blacks and Hispanics. 

H. A linkage of NAEP's 2002 reading assessment of 12 grade students. 

I. An assessment of the literacy skills of parents in NCES's Early Childhood Longitudinal 
Study in order to study the intergenerational transmission of literacy. 
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IV. Project Schedule 

The National Assessment of Adult Literacy in 2002 is expected to follow this schedule over 
the course of the project: 

❖ Planning will take place during 1998 and 1999. 

❖ Various stages of instrument development, field testing, sampling, interviewer training, 
data collection, and evaluation will take place prior to 2002. 

❖ Survey administration is expected to take place in 2002. 

❖ Data preparation, analysis, and initial reporting will begin in 2003. 

❖ Additional report preparation will be undertaken in 2004. 
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Please contact Angela Miles at (202) 219-1761 (angela_miles@ed.gov) 
if you are interested in any of the following papers 



Number 


Title 


Contact 


94-01 (July) 


Schools and Staffing Survey (SASS) Papers Presented 
at Meetings of the American Statistical Association 


Dan Kasprzyk 


94-02 (July) 


Generalized Variance Estimate for Schools and 
Staffing Survey (SASS) 


Dan Kasprzyk 


94-03 (July) 


1991 Schools and Staffing Survey (SASS) Reinterview 
Response Variance Report 


Dan Kasprzyk 


94-04 (July) 


The Accuracy of Teachers’ Self-reports on their 
Postsecondary Education: Teacher Transcript Study, 
Schools and Staffing Survey 


Dan Kasprzyk 


94-05 (July) 


Cost-of-Education Differentials Across the States 


William Fowler 


94-06 (July) 


Six Papers on Teachers from the 1990-91 Schools and 
Staffing Survey and Other Related Surveys 


Dan Kasprzyk 


94-07 (Nov.) 


Data Comparability and Public Policy: New Interest in 
Public Library Data Papers Presented at Meetings of 
the American Statistical Association 


Carrol Kindel 


95-01 (Jan.) 


Schools and Staffing Survey: 1994 Papers Presented at 
the 1994 Meeting of the American Statistical 
Association 


Dan Kasprzyk 


95-02 (Jan.) 


QED Estimates of the 1990-91 Schools and Staffing 
Survey: Deriving and Comparing QED School 
Estimates with CCD Estimates 


Dan Kasprzyk 


95-03 (Jan.) 


Schools and Staffing Survey: 1990-91 SASS Cross- 
Questionnaire Analysis 


Dan Kasprzyk 


95-04 (Jan.) 


National Education Longitudinal Study of 1988: 
Second Follow-up Questionnaire Content Areas and 
Research Issues 


Jeffrey Owings 


95-05 (Jan.) 


National Education Longitudinal Study of 1988: 
Conducting Trend Analyses of NLS-72, HS&B, and 
NELS:88 Seniors 


Jeffrey Owings 
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Number 


Title 


Contact 


95-06 (Jan.) 


National Education Longitudinal Study of 1988: 
Conducting Cross-Cohort Comparisons Using HS&B, 
NAEP, and NELS:88 Academic Transcript Data 


Jeffrey Owings 


95-07 (Jan.) 


National Education Longitudinal Study of 1988: 
Conducting Trend Analyses HS&B and NELS:88 
Sophomore Cohort Dropouts 


Jeffrey Owings 


95-08 (Feb.) 


CCD Adjustment to the 1990-91 SASS: A 
Comparison of Estimates 


Dan Kasprzyk 


95-09 (Feb.) 


The Results of the 1993 Teacher List Validation Study 
(TLVS) 


Dan Kasprzyk 


95-10 (Feb.) 


The Results of the 1991-92 Teacher Follow-up Survey 
(TFS) Reinterview and Extensive Reconciliation 


Dan Kasprzyk 


95-11 (Mar.) 


Measuring Instruction, Curriculum Content, and 
Instructional Resources: The Status of Recent Work 


Sharon Bobbitt & 
John Ralph 


95-12 (Mar.) 


Rural Education Data User’s Guide 


Samuel Peng 


95-13 (Mar.) 


Assessing Students with Disabilities and Limited 
English Proficiency 


James Houser 


95-14 (Mar.) 


Empirical Evaluation of Social, Psychological, & 
Educational Construct Variables Used in NCES 
Surveys 


Samuel Peng 


95-15 (Apr.) 


Classroom Instructional Processes: A Review of 
Existing Measurement Approaches and Their 
Applicability for the Teacher Follow-up Survey 


Sharon Bobbitt 


95-16 (Apr.) 


Intersurvey Consistency in NCES Private School 
Surveys 


Steven Kaufman 


95-17 (May) 


Estimates of Expenditures for Private K-12 Schools 


Stephen 

Broughman 


95-18 (Nov.) 


An Agenda for Research on Teachers and Schools: 
Revisiting NCES’ Schools and Staffing Survey 


Dan Kasprzyk 


96-01 (Jan.) 


Methodological Issues in the Study of Teachers’ 
Careers: Critical Features of a Truly Longitudinal 
Study 


Dan Kasprzyk 
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96-02 (Feb.) 

96-03 (Feb.) 

96-04 (Feb.) 
96-05 (Feb.) 

96-06 (Mar.) 

96-07 (Mar.) 
96-08 (Apr.) 
96-09 (Apr.) 

96-10 (Apr.) 
96-11 (June) 

96-12 (June) 

96-13 (June) 
96-14 (June) 



Schools and Staffing Survey (SASS): 1995 Selected 
papers presented at the 1995 Meeting of the American 
Statistical Association 

National Education Longitudinal Study of 1988 
(NELS:88) Research Framework and Issues 

Census Mapping Project/School District Data Book 

Cognitive Research on the Teacher Listing Form for 
the Schools and Staffing Survey 

The Schools and Staffing Survey (SASS) for 1998-99: 
Design Recommendations to Inform Broad Education 
Policy 

Should SASS Measure Instructional Processes and 
Teacher Effectiveness? 

How Accurate are Teacher Judgments of Students’ 
Academic Performance? 

Making Data Relevant for Policy Discussions: 
Redesigning the School Administrator Questionnaire 
for the 1998-99 SASS 

1998-99 Schools and Staffing Survey: Issues Related 
to Survey Depth 

Towards an Organizational Database on America’s 
Schools: A Proposal for the Future of SASS, with 
comments on School Reform, Governance, and 
Finance 

Predictors of Retention, Transfer, and Attrition of 
Special and General Education Teachers: Data from 
the 1989 Teacher Followup Survey 

Estimation of Response Bias in the NHES:95 Adult 
Education Survey 

The 1995 National Household Education Survey: 
Reinterview Results for the Adult Education 
Component 



Dan Kasprzyk 

Jeffrey Owings 

Tai Phan 
Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 
Jerry West 
Dan Kasprzyk 

Dan Kasprzyk 
Dan Kasprzyk 

Dan Kasprzyk 

Steven Kaufman 
Steven Kaufman 
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96-15 (June) 


Nested Structures: District-Level Data in the Schools 
and Staffing Survey 


Dan Kasprzyk 


96-16 (June) 


Strategies for Collecting Finance Data from Private 
Schools 


Stephen 

Broughman 


96-17 (July) 


National Postsecondary Student Aid Study: 1996 Field 
Test Methodology Report 


Andrew G. 
Malizio 


96-18 (Aug.) 


Assessment of Social Competence, Adaptive 
Behaviors, and Approaches to Learning with Young 
Children 


Jerry West 


96-19 (Oct.) 


Assessment and Analysis of School-Level 
Expenditures 


William Fowler 


96-20 (Oct.) 


1991 National Household Education Survey 
(NHES:91) Questionnaires: Screener, Early 
Childhood Education, and Adult Education 


Kathryn Chandler 


96-21 (Oct.) 


1 993 National Household Education Survey 
(NHES:93) Questionnaires: Screener, School 
Readiness, and School Safety and Discipline 


Kathryn Chandler 


96-22 (Oct.) 


1995 National Household Education Survey 
(NHES:95) Questionnaires: Screener, Early 
Childhood Program Participation, and Adult 
Education 


Kathryn Chandler 


96-23 (Oct.) 


Linking Student Data to SASS: Why, When, How 


Dan Kasprzyk 


96-24 (Oct.) 


National Assessments of Teacher Quality 


Dan Kasprzyk 


96-25 (Oct.) 


Measures of Inservice Professional Development: 
Suggested Items for the 1998-1999 Schools and 
Staffing Survey 


Dan Kasprzyk 


96-26 (Nov.) 


Improving the Coverage of Private Elementary- 
Secondary Schools 


Steven Kaufman 


96-27 (Nov.) 


Intersurvey Consistency in NCES Private School 
Surveys for 1993-94 


Steven Kaufman 
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96-28 (Nov.) 


Student Learning, Teaching Quality, and Professional 
Development: Theoretical Linkages, Current 
Measurement, and Recommendations for Future Data 
Collection 


Mary Rollefson 


96-29 (Nov.) 


Undercoverage Bias in Estimates of Characteristics of 
Adults and 0- to 2- Year-Olds in the 1995 National 
Household Education Survey (NHES:95) 


Kathryn Chandler 


96-30 (Dec.) 


Comparison of Estimates from the 1995 National 
Household Education Survey (NHES:95) 


Kathryn Chandler 


97-01 (Feb.) 


Selected Papers on Education Surveys: Papers 
Presented at the 1996 Meeting of the American 
Statistical Association 


Dan Kasprzyk 


97-02 (Feb.) 


Telephone Coverage Bias and Recorded Interviews in 
the 1993 National Household Education Survey 
(NHES:93) 


Kathryn Chandler 


97-03 (Feb.) 


1991 and 1995 National Household Education Survey 
Questionnaires: NHES:91 Screener, NHES:9l Adult 
Education, NHES:95 Basic Screener, and NHES:95 
Adult Education 


Kathryn Chandler 


97-04 (Feb.) 


Design, Data Collection, Monitoring, Interview 
Administration Time, and Data Editing in the 1993 
National Household Education Survey (NHES:93) 


Kathryn Chandler 


97-05 (Feb.) 


Unit and Item Response, Weighting, and Imputation 
Procedures in the 1993 National Household Education 
Survey (NHES.93) 


Kathryn Chandler 


97-06 (Feb.) 


Unit and Item Response, Weighting, and Imputation 
Procedures in the 1995 National Household Education 
Survey (NHES:95) 


Kathryn Chandler 


97-07 (Mar.) 


The Determinants of Per-Pupil Expenditures in 
Private Elementary and Secondary Schools: An 
Exploratory Analysis 


Stephen 

Broughman 


97-08 (Mar.) 


Design, Data Collection, Interview Timing, and Data 
Editing in the 1995 National Household Education 
Survey 


Kathryn Chandler 
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97-09 (Apr.) 


.Status of Data on Crime and Violence in Schools: 
Final Report 


Lee Hoffman 


97-10 (Apr.) 


Report of Cognitive Research on the Public and 
Private School Teacher Questionnaires for the Schools 
and Staffing Survey 1993-94 School Year 


Dan Kasprzyk 


97-11 (Apr.) 


International Comparisons of Inservice Professional 
Development 


Dan Kasprzyk 


97-12 (Apr.) 


Measuring School Reform: Recommendations for 
Future SASS Data Collection 


Mary Rollefson 


97-13 (Apr.) 


Improving Data Quality in NCES: Database-to-Report 
Process 


Susan Ahmed 


97-14 (Apr.) 


Optimal Choice of Periodicities for the Schools and 
Staffing Survey: Modeling and Analysis 


Steven Kaufman 


97-15 (May) 


Customer Service Survey: Common Core of Data 
Coordinators 


Lee Hoffman 


97-16 (May) 


International Education Expenditure Comparability 
Study: Final Report, Volume I 


Shelley Bums 


97-17 (May) 


International Education Expenditure Comparability 
Study: Final Report, Volume II, Quantitative Analysis 
of Expenditure Comparability 


Shelley Bums 


97-18 (June) 


Improving the Mail Return Rates of SASS Surveys: A 
Review of the Literature 


Steven Kaufman 


97-19 (June) 


National Household Education Survey of 1995: Adult 
Education Course Coding Manual 


Peter Stowe 


97-20 (June) 


National Household Education Survey of 1995: Adult 
Education Course Code Merge Files User’s Guide 


Peter Stowe 


97-21 (June) 


Statistics for Policymakers or Everything You Wanted 
to Know About Statistics But Thought You Could 
Never Understand 


Susan Ahmed 


97-22 (July) 


Collection of Private School Finance Data: 
Development of a Questionnaire 


Stephen 

Broughman 
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97-23 (July) 


Further Cognitive Research on the Schools and 
Staffing Survey (SASS) Teacher Listing Form 


Dan Kasprzyk 


97-24 (Aug.) 


Formulating a Design for the ECLS: A Review of 
Longitudinal Studies 


Jerry West 


97-25 (Aug.) 


1996 National Household Education Survey 
(NHES:96) Questionnaires: Screener/Household and 
Library, Parent and Family Involvement in Education 
and Civic Involvement, Youth Civic Involvement, and 
Adult Civic Involvement 


Kathryn Chandler 


97-26 (Oct.) 


Strategies for Improving Accuracy of Postsecondary 
Faculty Lists 


Linda Zimbler 


97-27 (Oct.) 


Pilot Test of IPEDS Finance Survey 


Peter Stowe 


97-28 (Oct.) 


Comparison of Estimates in the 1996 National 
Household Education Survey 


Kathryn Chandler 


97-29 (Oct.) 


Can State Assessment Data be Used to Reduce State 
NAEP Sample Sizes? 


Steven Gorman 


97-30 (Oct.) 


ACT’s NAEP Redesign Project: Assessment Design is 
the Key to Useful and Stable Assessment Results 


Steven Gorman 


97-31 (Oct.) 


NAEP Reconfigured: An Integrated Redesign of the 
National Assessment of Educational Progress 


Steven Gorman 


97-32 (Oct.) 


Innovative Solutions to Intractable Large Scale 
Assessment (Problem 2: Background Questionnaires) 


Steven Gorman 


97-33 (Oct.) 


Adult Literacy: An International Perspective 


Marilyn Binkley 


97-34 (Oct.) 


Comparison of Estimates from the 1993 National 
Household Education Survey 


Kathryn Chandler 


97-35 (Oct.) 


Design, Data Collection, Interview Administration 
Time, and Data Editing in the 1996 National 
Household Education Survey 


Kathryn Chandler 


97-36 (Oct.) 


Measuring the Quality of Program Environments in 
Head Start and Other Early Childhood Programs: A 
Review and Recommendations for Future Research 


Jerry West 
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97-37 (Nov.) 


Optimal Rating Procedures and Methodology for 
NAEP Open-ended Items 


Steven Gorman 


97-38 (Nov.) 


Reinterview Results for the Parent and Youth 
Components of the 1996 National Household 
Education Survey 


Kathryn Chandler 


97-39 (Nov.) 


Undercoverage Bias in Estimates of Characteristics of 
Households and Adults in the 1996 National 
Household Education Survey 


Kathryn Chandler 


97-40 (Nov.) 


Unit and Item Response Rates, Weighting, and 
Imputation Procedures in the 1996 National 
Household Education Survey 


Kathryn Chandler 


97-41 (Dec.) 


Selected Papers on the Schools and Staffing Survey: 
Papers Presented at the 1997 Meeting of the American 
Statistical Association 


Steve Kaufman 


97-42 
(Jan. 1998) 


Improving the Measurement of Staffing Resources at 
the School Level: The Development of 
Recommendations for NCES for the Schools and 
Staffing Survey (SASS) 


Mary Rollefson 


97-43 (Dec.) 


Measuring Inflation in Public School Costs 


William J. Fowler, 
Jr. 


97-44 (Dec.) 


Development of a SASS 1993-94 School-Level 
Student Achievement Subfile: Using State 
Assessments and State NAEP, Feasibility Study 


Michael Ross 


98-01 (Jan.) 


Collection of Public School Expenditure Data: 
Development of a Questionnaire 


Stephen 

Broughman 


98-02 (Jan.) 


Response Variance in the 1993-94 Schools and 
Staffing Survey: A Reinterview Report 


Steven Kaufman 


98-03 (Feb.) 


Adult Education in the 1990s: A Report on the 1991 
National Household Education Survey 


Peter Stowe 


98-04 (Feb.) 


Geographic Variations in Public Schools’ Costs 


William J. Fowler, 
Jr. 
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98-05 (Mar.) 


SASS Documentation: 1993-94 SASS Student 
Sampling Problems; Solutions for Determining the 
Numerators for the SASS Private School (3B) 
Second-Stage Factors 


Steven Kaufman 


98-06 (May) 


National Education Longitudinal Study of 1988 
(NELS:88) Base Year through Second Follow-Up: 
Final Methodology Report 


Ralph Lee 


98-07 (May) 


Decennial Census School District Project Planning 
Report 


Tai Phan 


98-08 (July) 


The Redesign of the Schools and Staffing Survey for 
1999-2000: A Position Paper 


Dan Kasprzyk 


98-09 (Aug.) 


High School Curriculum Structure: Effects on 
Coursetaking and Achievement in Mathematics for 
High School Graduates — An Examination of Data 
from the National Education Longitudinal Study of 
1988 


Jeffrey Owings 


98-10 (Aug.) 


Adult Education Participation Decisions and Barriers: 
Review of Conceptual Frameworks and Empirical 
Studies 


Peter Stowe 


98-1 1 (Aug.) 


Beginning Postsecondary Students Longitudinal Study 
First Follow-up (BPS:96-98) Field Test Report 


Aurora D’Amico 


98-12 (Oct.) 


A Bootstrap Variance Estimator for Systematic PPS 
Sampling 


Steven Kaufman 


98-13 (Oct.) 


Response Variance in the 1994-95 Teacher Follow-up 
Survey 


Steven Kaufman 


98-14 (Oct.) 


Variance Estimation of Imputed Survey Data 


Steven Kaufman 


98-15 (Oct.) 


Development of a Prototype System for Accessing 
Linked NCES Data 


Steven Kaufman 


98-16 (Dec.) 


A Feasibility Study of Longitudinal Design for 
Schools and Staffing Survey 


Stephen 

Broughman 


98-17 (Dec.) 


Developing the National Assessment of Adult 
Literacy: Recommendations from Stakeholders 


Sheida White 
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